Computing devices continue to become more ubiquitous to daily life. They take the form of computer desktops, laptop computers, tablet computers, hybrid computers (2-in-1s), e-book readers, mobile phones, smartphones, wearable computers, global positioning system (GPS) units, enterprise digital assistants (EDAs), personal digital assistants (PDAs), game consoles, and the like. Further, computing devices are being incorporated into vehicles and equipment, such as cars, trucks, farm equipment, manufacturing equipment, building environment control (e.g., lighting, HVAC), and home and commercial appliances.
Computing devices generally consist of at least one processing element, such as a central processing unit (CPU), some form of memory, and input and output devices. The variety of computing devices and their subsequent uses necessitate a variety of interfaces and input devices. One such input device is a touch sensitive surface such as a touch screen or touch pad wherein user input is received through contact between the user's finger or an instrument such as a pen or stylus and the touch sensitive surface. Another input device is an input surface that senses gestures made by a user above the input surface. A further input device is a position detection system which detects the relative position of either touch or non-touch interactions with a non-touch physical or virtual surface. Any of these methods of input can be used generally for drawing or inputting text. The user's handwriting is interpreted using a handwriting recognition system or method.
One application of handwriting recognition in portable computing devices, such as smartphones, phablets and tablets, is in note taking. This particularly occurs in education and business settings where the user of the computing device captures notes, for example, during a lecture or meeting. This is usually done by the user launching a handwritten note taking application on the computing device which accepts and interprets, either locally in the device or remotely via a communications link of the device, handwritten notes input on the touch sensitive surface. Conventionally such handwritten note taking applications are limited in their capabilities to provide a full document creation experience to users from the notes taken, since the focus of these applications has primarily been recognition accuracy rather than document creation. That is, available applications provide recognition of handwriting and conversion of the recognized handwriting into typeset text, with various feedback mechanisms to the user, but that is generally the extent of the interaction with the input.
If the user desires any further interaction with the output text, such as editing the content, manipulating the layout of the notes, or converting or adding the notes into a document, the text generally needs to be imported into a separate document processing application. This in itself is no great problem, however as the original layout of the handwritten notes and the actual input handwriting itself, so-called digital ink, is typically discarded in the import process, the user must refer back to the original notes in order to ascertain the intent in the notes taken. For example, the user may have emphasized certain words or passages in the notes either by annotation or decoration, or through the layout of the notes content itself.
Further, once the import is performed into the separate application the ability to add or edit content through further note taking is generally lost, and the user must in effect begin a new note which must itself be imported into the document. Such a cumbersome method clearly limits the ability of continuing or updating notes for inclusion in documents over different sessions, e.g., over multiple lectures or multiple business meetings, or through collaboration and sharing. Of course, users could delay importation to a document processing application until the note taking is finished over such multiple sessions, with or without multiple users. However, since many people nowadays have multiple portable computing devices which are interconnected it may be desired to perform these multiple sessions on these different computing devices, or at least access the notes on these different devices.
For example, a user may begin a note on their laptop during a lecture or meeting, then wish to add to that note using their tablet during a conversation with a colleague or friend, and then later share the updated note using their smartphone. The available input and display space on the screens of these devices can vary significantly. Further, some devices allow display at different orientations, i.e., portrait and landscape. As such, the user's layout of the notes using one device at one orientation may be significantly changed when the notes are displayed and interacted with on a different device or at a different orientation. Accordingly, continued note taking on specific note layout over these different display scenarios is presently difficult.
Other issues with the note taking layout occur without even considering the complexities of multiple devices and orientations. Available handwritten note taking applications generally provide users with the ability to typeset the handwritten input either manually or automatically (e.g., on-the-fly). However, as typeset text is generally smaller and more uniform than handwritten text (described in detail later), the user's layout of the handwritten input, such as in sentences in paragraphs, may not translate well to typeset text, e.g., the sentences become too short such that the paragraphs lose visual meaning and look more like lists than paragraphed text. This can be overcome by reflowing the typeset text so that the layout appears to be preserved, e.g., paragraphs are retained. However, the user's original handwritten layout may have actually been a list not a paragraph, for example, and without an explicit indication of such layouts it is difficult for the handwriting recognition process to detect these instances.
Conventional processes require users to input certain symbols or characters, e.g., bullet points for list input (see, for example, U.S. Pat. No. 5,613,019), or use particular gestures to indicate such layouts, or implement input methods for such which otherwise disturb the handwriting flow of the user (see, for example, U.S. Pat. No. 7,751,623 which discloses display of a guide for list input). However, the user should be able to input handwritten notes in their desired layout without needing to consider the manner in which the handwritten note taking application will handle that input. This is because, the Applicant has found that when using handwriting note taking applications users generally are unable or do not desire to learn specific input methods or gestures that are not natural or intuitive, or to make settings through menus and the like.
Further, note taking generally takes many forms, and user's don't only take notes using text characters. For example, notes may include many types of content, such as text, drawings, diagrams, mathematical or chemical equations and music notation. Further, the user may want to insert other media into the note, such as images and audio or video recordings. Accordingly, such multi-content and multi-media input should be supported whilst respecting the user's layout of these inputs.
The user's note layout however should be made in a fashion that assists in the effective recognition of the handwritten content as well as the sensible typesetting of that content. Some conventional systems provide assistance for recognition by guiding the handwritten input through use of ruled lines, similar to paper. U.S. Pat. Nos. 5,613,019, 7,096,432, 7,167,165 and 7,266,769 disclose note taking systems with such lines provided in various forms. With respect to the line height, e.g., the space between the lines, U.S. Pat. No. 7,167,165 discusses setting it based on the size of the handwritten input, U.S. Pat. No. 7,266,769 discusses making it user settable and U.S. Pat. No. 5,613,019 discusses setting it in a manner which allows discrimination between handwritten text input and drawing input. Such considerations do not take the eventual typesetting of the handwritten text into account however and therefore the set spacing and distribution of the lines may be made independently of the typeset ink which is not preferable for sensible typesetting of the handwritten note content and layout.